{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 问题描述\n",
    "\n",
    "数据：Million Song Dataset(MSD)\n",
    "\n",
    "https://labrosa.ee.columbia.edu/millionsong/\n",
    "    \n",
    "作业使用的数据集是公开音乐数据集 Million Song Dataset(MSD) ， 它 包 含 来 自 SecondHandSongs dataset 、 musiXmatch dataset、Last.fm dataset、Taste Profile subset、 thisismyjam-to-MSD mapping、tagtraum genre annotations 和 Top MAGD dataset 七个知名音乐社区的数据。\n",
    "\n",
    "原始数据集包括：\n",
    "\n",
    "1. train_triplets.txt：三元组数据（用户、歌曲、播放次数）\n",
    "\n",
    "2. track_metadata.db：每个歌曲的元数据 \n",
    "\n",
    "由于原始数据太大，作业用的数据集只是其中的子集（播放次数最多的10万个用户、播放次数最多的3万首歌曲。 \n",
    "数据预处理过程请见DataProcessing.ipynb文件，最后得到的数据文件为：triplet_dataset_sub_song_merged.csv（1千万条记录） "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 解题提示\n",
    "\n",
    "注意：\n",
    "\n",
    "1. 如果内存有限，对每个用户，可考虑只推荐最流行的5000首歌曲。从triplet_dataset_sub_song_merged.csv中抽取播放次数排在前5000的歌曲。Recommendation_Item_CF.ipynb文件中已有抽取数据的代码。\n",
    "\n",
    "2. 由于这个数据集中并没有用户对物品的显式打分，需要将播放次数转换为分数。 \n",
    "\n",
    "3. 在协同过滤中计算用户之间的相似度或物品之间的相似度时，一种方式用播放次数/分数作为用户/物品的特征表示，同课件。可考虑将播放次数变换到[0,10]区间。 \n",
    "另一种可选的表示是只要用户播放过歌曲就表示为1，否则为0（二值化），这样物品之间的相似度为播放两个歌曲的用户交集除以播放两个歌曲的用户并集： \n",
    "。 类似的，两个用户之间的相似度可用两个用户播放歌曲的交集除以两个用户播放歌曲的并集表示。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 批改标准\n",
    "\n",
    "作业要求：将triplet_dataset_sub.csv中的数据用train_test_split分成60%数据做训练，剩下40%数据做测试。\n",
    "\n",
    "1. 实现基于用户的协同过滤； （20分）\n",
    "\n",
    "2. 实现基于物品的协同过滤； （20分）\n",
    "\n",
    "3. 实现基于模型（矩阵分解/LFM）的协同过滤。（30分）\n",
    "\n",
    "4. 对每种推荐算法的推荐结果，用Top20个推荐歌曲的准确率和召回率评价推荐系统的性能。（30分）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>song</th>\n",
       "      <th>play_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4e11f45d732f4861772b2906f81a7d384552ad12</td>\n",
       "      <td>SOCKSGZ12A58A7CA4B</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4e11f45d732f4861772b2906f81a7d384552ad12</td>\n",
       "      <td>SOCVTLJ12A6310F0FD</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4e11f45d732f4861772b2906f81a7d384552ad12</td>\n",
       "      <td>SODLLYS12A8C13A96B</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4e11f45d732f4861772b2906f81a7d384552ad12</td>\n",
       "      <td>SOEGIYH12A6D4FC0E3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4e11f45d732f4861772b2906f81a7d384552ad12</td>\n",
       "      <td>SOFRQTD12A81C233C0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       user                song  play_count\n",
       "0  4e11f45d732f4861772b2906f81a7d384552ad12  SOCKSGZ12A58A7CA4B           1\n",
       "1  4e11f45d732f4861772b2906f81a7d384552ad12  SOCVTLJ12A6310F0FD           1\n",
       "2  4e11f45d732f4861772b2906f81a7d384552ad12  SODLLYS12A8C13A96B           3\n",
       "3  4e11f45d732f4861772b2906f81a7d384552ad12  SOEGIYH12A6D4FC0E3           1\n",
       "4  4e11f45d732f4861772b2906f81a7d384552ad12  SOFRQTD12A81C233C0           2"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读入数据\n",
    "df_data = pd.read_csv('triplet_dataset_sub.csv', encoding='latin-1')\n",
    "\n",
    "df_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 将 user 和 song 的字符串转化成数字"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "le = preprocessing.LabelEncoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# User 数字化后的小字典\n",
    "le.fit(df_data['user'].values)\n",
    "user = le.transform(df_data['user'].values) \n",
    "df_data_user = pd.DataFrame(data=user, columns=['user'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Song 数字化后的小字典\n",
    "le.fit(df_data['song'].values)\n",
    "song = le.transform(df_data['song'].values) \n",
    "df_data_song = pd.DataFrame(data=song, columns=['song'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将 play_count 映射到区间 [1,5]\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "mm = MinMaxScaler(feature_range=(1,5))\n",
    "col_label = ['play_count']\n",
    "mm_df_data = mm.fit_transform(df_data[col_label])\n",
    "\n",
    "# play_count 的小字典\n",
    "df_data_count = pd.DataFrame(data=mm_df_data, columns=['rating'], index =df_data.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>song</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>246</td>\n",
       "      <td>4</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>246</td>\n",
       "      <td>5</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>246</td>\n",
       "      <td>10</td>\n",
       "      <td>1.002266</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>246</td>\n",
       "      <td>12</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>246</td>\n",
       "      <td>23</td>\n",
       "      <td>1.001133</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  song    rating\n",
       "0   246     4  1.000000\n",
       "1   246     5  1.000000\n",
       "2   246    10  1.002266\n",
       "3   246    12  1.000000\n",
       "4   246    23  1.001133"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Join dics\n",
    "df_data = pd.concat([df_data_user, df_data_song, df_data_count], axis = 1, ignore_index=False)\n",
    "df_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 37519 entries, 0 to 37518\n",
      "Data columns (total 3 columns):\n",
      "user      37519 non-null int64\n",
      "song      37519 non-null int64\n",
      "rating    37519 non-null int64\n",
      "dtypes: int64(3)\n",
      "memory usage: 879.4 KB\n"
     ]
    }
   ],
   "source": [
    "df_data_int = df_data.astype('int64')\n",
    "df_data = df_data_int\n",
    "df_data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>song</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>246</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>246</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>246</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>246</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>246</td>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  song  rating\n",
       "0   246     4       1\n",
       "1   246     5       1\n",
       "2   246    10       1\n",
       "3   246    12       1\n",
       "4   246    23       1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 测试集训练集分离"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_data, test_data = train_test_split(df_data, test_size = 0.40, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 基于模型（SVD）的协同过滤推荐"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 用户、物品数去重统计\n",
    "n_users = df_data.user.nunique()                   \n",
    "n_items = df_data.song.nunique() \n",
    "\n",
    "# 训练集 用户-物品矩阵\n",
    "train_data_matrix = np.zeros((n_users, n_items))      \n",
    "for line in train_data.itertuples():\n",
    "    train_data_matrix[line[1]-1, line[2]-1] = line[3] \n",
    "\n",
    "# 测试集 用户-物品矩阵 \n",
    "test_data_matrix = np.zeros((n_users, n_items))       \n",
    "for line in test_data.itertuples():\n",
    "    test_data_matrix[line[1]-1, line[2]-1] = line[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVD分解"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy.sparse as sp\n",
    "from scipy.sparse.linalg import svds\n",
    "\n",
    "u, s, vt = svds(train_data_matrix, k = 20)     #获取奇异值分解因子。选择K.\n",
    "s_diag_matrix=np.diag(s)\n",
    "svd_prediction = np.dot(np.dot(u, s_diag_matrix), vt) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "svd_prediction[svd_prediction<0] = 0       # 将预测值中小于0的值，赋值为0\n",
    "svd_prediction[svd_prediction>5] = 5       # 将预测值中大于5的值，赋值为5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练 与 预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7404286800760769"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 训练集预测\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from math import sqrt\n",
    "\n",
    "# 只取预测数据集中有评分的数据集，进行评估\n",
    "prediction_flatten = svd_prediction[train_data_matrix.nonzero()] \n",
    "train_data_matrix_flatten = train_data_matrix[train_data_matrix.nonzero()]\n",
    "sqrt(mean_squared_error(prediction_flatten, train_data_matrix_flatten)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8497714468844566"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 测试集预测\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from math import sqrt\n",
    "\n",
    "# 只取训练数据集中有评分的数据集，进行评估\n",
    "prediction_flatten = svd_prediction[test_data_matrix.nonzero()] \n",
    "test_data_matrix_flatten = test_data_matrix[test_data_matrix.nonzero()]\n",
    "sqrt(mean_squared_error(prediction_flatten, test_data_matrix_flatten))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Top20个推荐歌曲的准确率和召回率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 只选择在训练集没有被评过分的部分进行推荐 \n",
    "svd_p_df = pd.DataFrame(svd_prediction)\n",
    "svd_p_df = svd_p_df[pd.DataFrame(train_data_matrix) == 0] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 利用分位数函数获取每个用户的得分最高的topk个物品\n",
    "topk = 20                                    # 看top20的相似\n",
    "quantile = 1-topk/float(svd_p_df.shape[1])   # topk的分位数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 预测矩阵的topk 0,1矩阵\n",
    "# 每个用户的topk分位数值\n",
    "# 如果topk分位数值为0,则意味此行大于0的列数<topk个，则为了后面的处理需要取一个比0稍微大一点点的值\n",
    "topk_threshold = svd_p_df.quantile(q=quantile, axis=1).map(lambda x: max(x, 0.00000000000001))\n",
    "# 对每行中前topk的数值标记为1，否则标记为0\n",
    "svd_p_topk = svd_p_df.sub(topk_threshold, axis=0).applymap(lambda x:1 if x>=0 else 0) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 测试矩阵的topk 0,1矩阵\n",
    "test_data_m_df = pd.DataFrame(test_data_matrix)\n",
    "# 每个用户的topk分位数值\n",
    "topk_threshold = test_data_m_df.quantile(q=quantile, axis=1).map(lambda x: max(x, 0.000000001))\n",
    "# 对每行中前topk的数值标记为1，否则标记为0\n",
    "test_data_m_topk = test_data_m_df.sub(topk_threshold, axis=0).applymap(lambda x:1 if x>=0 else 0) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 并集矩阵\n",
    "inter_m = test_data_m_topk*svd_p_topk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              recall           precision\n",
      "             17.870%             17.494%\n"
     ]
    }
   ],
   "source": [
    "# 在测试集上给出Top20个推荐歌曲的准确率和召回率\n",
    "print(\"%20s%20s\" % ('recall','precision'))\n",
    "precision = inter_m.sum().sum() / np.double(svd_p_topk.sum().sum())\n",
    "recall = inter_m.sum().sum() / np.double(test_data_m_topk.sum().sum())\n",
    "print(\"%19.3f%%%19.3f%%\" % (recall * 100,precision * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**下面表格总结了三种模型的准确率和召回率:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              method              recall           precision\n",
      "                User             11.326%             10.885%\n",
      "                Item             12.653%             12.160%\n",
      "                 SVD             17.870%             17.494%\n"
     ]
    }
   ],
   "source": [
    "print(\"%20s%20s%20s\" % ('method','recall','precision'))\n",
    "print(\"%20s%19.3f%%%19.3f%%\" % ('User',11.326,10.885))\n",
    "print(\"%20s%19.3f%%%19.3f%%\" % ('Item',12.653,12.160))\n",
    "print(\"%20s%19.3f%%%19.3f%%\" % ('SVD',17.870,17.494))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**以上准确率和召回率的结果显示：**\n",
    "\n",
    "基于模型的协同过滤 优于 基于物品的协同过滤 优于 基于用户的协同过滤。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
